05. Measuring Outcomes

Measuring Outcomes

The goals of your study may not be the same as the way you evaluate the study's
success. Perhaps this is because the goal is something that can't be measured
directly. Let's say that you have an idea of a website addition that improves
user satisfaction. How should we measure this? In order to evaluate whether
or not this improvement has happened, you need to have a way to objectively
measure the effect of the addition. For example, you might include a survey to
random users to have them rate their website experience on a 1-10 scale. If the
addition is helpful, then we should expect the average rating to be higher for
those users who are given the addition, versus those who are not. The rating
scale acts as a concrete way of measuring user satisfaction. These objective
features by which you evaluate performance are known as evaluation metrics.

As a rule of thumb, it's a good idea to consider the goals of a study separate
from the evaluation metrics. This provides a couple of useful benefits. First,
this makes it clear that the metric isn't the main point of a study: it's the
implications of the metric relative to the goal that matters. This is
especially important if a metric isn't directly attached to the goal. For
example, measuring students' confidence going into a standardized test might be
a proxy for the goal of test preparedness, in the absence of being able to get
their test scores directly or in a timely fashion.

Secondly, having the metric separate from the goal can clarify the purpose of
conducting the study or experiment. It makes sure we can answer the question of
why we want to run a study or experiment. From the above example, we aren't
measuring confidence just to make people feel good about themselves: we're
doing it to try and improve their actual performances.

Side Note: Alternate Terminology

You might hear other terminology for goals and evaluation metrics than those
used in this course. In the social sciences, it's common to hear a "construct"
as analogous to the goal or objective under investigation, and the "operational
definition" as the way outcomes are measured. For example, the construct of
"reaction time" could be operationally defined as "time in milliseconds to
click on the correctly indicated button."

In general company operations, you might encounter the terms "key results"
(KRs) or "key performance indicators" (KPIs) as ways of measuring progress
against quarterly or annual "objectives." These objectives and KRs / KPIs serve
a similar purpose as study goals and evaluation metrics, and might even be
driving factors in the creation of an experiment.

References

Inspiration for the "number of searches" metric shown in the video above came from this Hackernoon article on the Bing search engine.